chore(NET-112): Add http api, streams and query related metrics to HotblockDB #48

define-null · 2026-01-14T07:55:19Z

What is this PR about?

Closes: https://linear.app/sqd-ai/issue/NET-112/add-metrics-to-hotblocksdb

PR adds metrics for main http endpoints, as well as stream/query related information for HotblocksDB.

Metrics example from the local run

# HELP hotblocks_first_block First block
# TYPE hotblocks_first_block gauge
hotblocks_first_block{dataset="my-devnet"} 24228465
# HELP hotblocks_last_block Last block
# TYPE hotblocks_last_block gauge
hotblocks_last_block{dataset="my-devnet"} 24228719
# HELP hotblocks_last_block_timestamp_ms Timestamp of the last block
# TYPE hotblocks_last_block_timestamp_ms gauge
hotblocks_last_block_timestamp_ms{dataset="my-devnet"} 1768342763000
# HELP hotblocks_last_finalized_block Last finalized block
# TYPE hotblocks_last_finalized_block gauge
hotblocks_last_finalized_block{dataset="my-devnet"} 24228632
# HELP hotblocks_query_error_too_many_tasks Number of query tasks rejected due to task queue overflow.
# TYPE hotblocks_query_error_too_many_tasks counter
hotblocks_query_error_too_many_tasks_total 0
# HELP hotblocks_query_error_too_many_data_waiters Number of queries rejected, because data is not yet available and there are too many data waiters.
# TYPE hotblocks_query_error_too_many_data_waiters counter
hotblocks_query_error_too_many_data_waiters_total 0
# HELP hotblocks_http_status Number of sent HTTP responses.
# TYPE hotblocks_http_status counter
hotblocks_http_status_total{dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_status_total{dataset_id="my-devnet",endpoint="/stream",status="400"} 1
# HELP hotblocks_http_seconds_to_first_byte Time to first byte of HTTP responses.
# TYPE hotblocks_http_seconds_to_first_byte histogram
hotblocks_http_seconds_to_first_byte_sum{dataset_id="my-devnet",endpoint="/stream",status="200"} 0.024065875
hotblocks_http_seconds_to_first_byte_count{dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="0.001",dataset_id="my-devnet",endpoint="/stream",status="200"} 0
hotblocks_http_seconds_to_first_byte_bucket{le="0.0015",dataset_id="my-devnet",endpoint="/stream",status="200"} 0
hotblocks_http_seconds_to_first_byte_bucket{le="0.0025",dataset_id="my-devnet",endpoint="/stream",status="200"} 0
hotblocks_http_seconds_to_first_byte_bucket{le="0.005",dataset_id="my-devnet",endpoint="/stream",status="200"} 0
hotblocks_http_seconds_to_first_byte_bucket{le="0.01",dataset_id="my-devnet",endpoint="/stream",status="200"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.015",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="0.025",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="0.05",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="0.1",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="0.15000000000000003",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="0.25",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="0.5",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="1.0",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="1.5",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="2.5",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="5.0",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="10.0",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="15.0",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="25.0",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="50.0",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_bucket{le="+Inf",dataset_id="my-devnet",endpoint="/stream",status="200"} 2
hotblocks_http_seconds_to_first_byte_sum{dataset_id="my-devnet",endpoint="/stream",status="400"} 0.00016325
hotblocks_http_seconds_to_first_byte_count{dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.001",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.0015",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.0025",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.005",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.01",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.015",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.025",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.05",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.1",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.15000000000000003",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.25",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="0.5",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="1.0",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="1.5",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="2.5",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="5.0",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="10.0",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="15.0",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="25.0",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="50.0",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
hotblocks_http_seconds_to_first_byte_bucket{le="+Inf",dataset_id="my-devnet",endpoint="/stream",status="400"} 1
# HELP hotblocks_stream_bytes Numbers of bytes per stream.
# TYPE hotblocks_stream_bytes histogram
hotblocks_stream_bytes_sum{dataset_id="my-devnet"} 21581.0
hotblocks_stream_bytes_count{dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="1000.0",dataset_id="my-devnet"} 0
hotblocks_stream_bytes_bucket{le="2000.0",dataset_id="my-devnet"} 0
hotblocks_stream_bytes_bucket{le="4000.0",dataset_id="my-devnet"} 0
hotblocks_stream_bytes_bucket{le="8000.0",dataset_id="my-devnet"} 0
hotblocks_stream_bytes_bucket{le="16000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="32000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="64000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="128000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="256000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="512000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="1024000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="2048000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="4096000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="8192000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="16384000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="32768000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="65536000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="131072000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="262144000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="524288000.0",dataset_id="my-devnet"} 2
hotblocks_stream_bytes_bucket{le="+Inf",dataset_id="my-devnet"} 2
# HELP hotblocks_stream_blocks Numbers of blocks per stream.
# TYPE hotblocks_stream_blocks histogram
hotblocks_stream_blocks_sum{dataset_id="my-devnet"} 510.0
hotblocks_stream_blocks_count{dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="1.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_bucket{le="2.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_bucket{le="4.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_bucket{le="8.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_bucket{le="16.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_bucket{le="32.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_bucket{le="64.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_bucket{le="128.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_bucket{le="256.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="512.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="1024.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="2048.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="4096.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="8192.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="16384.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="32768.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="65536.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="131072.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="262144.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="524288.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="1048576.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="2097152.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="4194304.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="8388608.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="16777216.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="33554432.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="67108864.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="134217728.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="268435456.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="536870912.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_bucket{le="+Inf",dataset_id="my-devnet"} 2
# HELP hotblocks_stream_chunks Numbers of chunks per stream.
# TYPE hotblocks_stream_chunks histogram
hotblocks_stream_chunks_sum{dataset_id="my-devnet"} 12.0
hotblocks_stream_chunks_count{dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="1.0",dataset_id="my-devnet"} 0
hotblocks_stream_chunks_bucket{le="1.5",dataset_id="my-devnet"} 0
hotblocks_stream_chunks_bucket{le="2.5",dataset_id="my-devnet"} 0
hotblocks_stream_chunks_bucket{le="5.0",dataset_id="my-devnet"} 0
hotblocks_stream_chunks_bucket{le="10.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="15.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="25.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="50.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="100.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="150.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="250.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="500.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="1000.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="1500.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="2500.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="5000.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="10000.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="15000.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="25000.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="50000.0",dataset_id="my-devnet"} 2
hotblocks_stream_chunks_bucket{le="+Inf",dataset_id="my-devnet"} 2
# HELP hotblocks_stream_bytes_per_second Completed streams bandwidth.
# TYPE hotblocks_stream_bytes_per_second histogram
hotblocks_stream_bytes_per_second_sum 1853834.8436018534
hotblocks_stream_bytes_per_second_count 2
hotblocks_stream_bytes_per_second_bucket{le="100.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="300.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="900.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="2700.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="8100.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="24300.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="72900.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="218700.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="656100.0"} 0
hotblocks_stream_bytes_per_second_bucket{le="1968300.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="5904900.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="17714700.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="53144100.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="159432300.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="478296900.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="1434890700.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="4304672100.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="12914016300.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="38742048900.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="116226146700.0"} 2
hotblocks_stream_bytes_per_second_bucket{le="+Inf"} 2
# HELP hotblocks_stream_blocks_per_second Completed streams speed in blocks.
# TYPE hotblocks_stream_blocks_per_second histogram
hotblocks_stream_blocks_per_second_sum{dataset_id="my-devnet"} 44493.63863694247
hotblocks_stream_blocks_per_second_count{dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="1.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="3.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="9.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="27.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="81.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="243.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="729.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="2187.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="6561.0",dataset_id="my-devnet"} 0
hotblocks_stream_blocks_per_second_bucket{le="19683.0",dataset_id="my-devnet"} 1
hotblocks_stream_blocks_per_second_bucket{le="59049.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="177147.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="531441.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="1594323.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="4782969.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="14348907.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="43046721.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="129140163.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="387420489.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="1162261467.0",dataset_id="my-devnet"} 2
hotblocks_stream_blocks_per_second_bucket{le="+Inf",dataset_id="my-devnet"} 2
# HELP hotblocks_stream_duration_seconds Durations of completed streams.
# TYPE hotblocks_stream_duration_seconds histogram
hotblocks_stream_duration_seconds_sum{dataset_id="my-devnet"} 0.024054166000000003
hotblocks_stream_duration_seconds_count{dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="0.01",dataset_id="my-devnet"} 1
hotblocks_stream_duration_seconds_bucket{le="0.02",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="0.04",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="0.08",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="0.16",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="0.32",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="0.64",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="1.28",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="2.56",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="5.12",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="10.24",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="20.48",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="40.96",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="81.92",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="163.84",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="327.68",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="655.36",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="1310.72",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="2621.44",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="5242.88",dataset_id="my-devnet"} 2
hotblocks_stream_duration_seconds_bucket{le="+Inf",dataset_id="my-devnet"} 2
# HELP hotblocks_queried_bytes Numbers of bytes queried per running query.
# TYPE hotblocks_queried_bytes histogram
hotblocks_queried_bytes_sum{dataset_id="my-devnet"} 21581.0
hotblocks_queried_bytes_count{dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="1000.0",dataset_id="my-devnet"} 0
hotblocks_queried_bytes_bucket{le="2000.0",dataset_id="my-devnet"} 0
hotblocks_queried_bytes_bucket{le="4000.0",dataset_id="my-devnet"} 0
hotblocks_queried_bytes_bucket{le="8000.0",dataset_id="my-devnet"} 0
hotblocks_queried_bytes_bucket{le="16000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="32000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="64000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="128000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="256000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="512000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="1024000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="2048000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="4096000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="8192000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="16384000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="32768000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="65536000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="131072000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="262144000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="524288000.0",dataset_id="my-devnet"} 2
hotblocks_queried_bytes_bucket{le="+Inf",dataset_id="my-devnet"} 2
# HELP hotblocks_queried_blocks Numbers of blocks per running query.
# TYPE hotblocks_queried_blocks histogram
hotblocks_queried_blocks_sum{dataset_id="my-devnet"} 510.0
hotblocks_queried_blocks_count{dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="1.0",dataset_id="my-devnet"} 0
hotblocks_queried_blocks_bucket{le="2.0",dataset_id="my-devnet"} 0
hotblocks_queried_blocks_bucket{le="4.0",dataset_id="my-devnet"} 0
hotblocks_queried_blocks_bucket{le="8.0",dataset_id="my-devnet"} 0
hotblocks_queried_blocks_bucket{le="16.0",dataset_id="my-devnet"} 0
hotblocks_queried_blocks_bucket{le="32.0",dataset_id="my-devnet"} 0
hotblocks_queried_blocks_bucket{le="64.0",dataset_id="my-devnet"} 0
hotblocks_queried_blocks_bucket{le="128.0",dataset_id="my-devnet"} 0
hotblocks_queried_blocks_bucket{le="256.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="512.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="1024.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="2048.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="4096.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="8192.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="16384.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="32768.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="65536.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="131072.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="262144.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="524288.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="1048576.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="2097152.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="4194304.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="8388608.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="16777216.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="33554432.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="67108864.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="134217728.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="268435456.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="536870912.0",dataset_id="my-devnet"} 2
hotblocks_queried_blocks_bucket{le="+Inf",dataset_id="my-devnet"} 2
# HELP hotblocks_queried_chunks Numbers of chunks per running query.
# TYPE hotblocks_queried_chunks histogram
hotblocks_queried_chunks_sum{dataset_id="my-devnet"} 12.0
hotblocks_queried_chunks_count{dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="1.0",dataset_id="my-devnet"} 0
hotblocks_queried_chunks_bucket{le="1.5",dataset_id="my-devnet"} 0
hotblocks_queried_chunks_bucket{le="2.5",dataset_id="my-devnet"} 0
hotblocks_queried_chunks_bucket{le="5.0",dataset_id="my-devnet"} 0
hotblocks_queried_chunks_bucket{le="10.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="15.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="25.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="50.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="100.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="150.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="250.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="500.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="1000.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="1500.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="2500.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="5000.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="10000.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="15000.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="25000.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="50000.0",dataset_id="my-devnet"} 2
hotblocks_queried_chunks_bucket{le="+Inf",dataset_id="my-devnet"} 2
# HELP hotblocks_active_queries Number of active queries.
# TYPE hotblocks_active_queries gauge
hotblocks_active_queries 0
# HELP hotblocks_completed_queries Number of completed queries.
# TYPE hotblocks_completed_queries counter
hotblocks_completed_queries_total 3
# EOF

Some of the style changes may appear excessive due to rustfmt not having been run previously.

…tblockDB

kalabukdima · 2026-01-14T12:08:27Z

crates/hotblocks/src/query/executor.rs

+    pub fn spawn_metrics_reporter(&self, interval: Duration) {
+        let active_count = self.in_flight.clone();
+
+        tokio::spawn(async move {


Why not just increase/decrease this one every time when the active_count gets updated?
Also what if there are multiple query executors in the process?

Why not just increase/decrease this one every time when the active_count gets updated?

I agree that’s possible. The reasoning here: we already have one shared atomic (active_count) that’s updated in a few places and has guardrails/clamping to keep it within bounds. If we also manually increment/decrement a separate metric counter in the same places, that’s extra duplication and increases the chance the metric drifts from the actual value over time (especially if another update site gets added later).

With the current approach, the metric just reflects the authoritative value after it changes. Can you clarify your concern?

Also what if there are multiple query executors in the process?

From what I can tell the code isn’t currently structured/documented to support multiple QueryExecutors. And even if we did have multiple instances, sqd_polars::POOL is shared across them anyway, so it’s not clear that multiple executors would give isolation or change the semantics of the metric much.

Are you expecting a multi-executor setup where each instance should report separate metrics (per executor), or is the intent a single process-wide metric?

crates/hotblocks/src/api.rs

Copilot

Pull request overview

This PR adds comprehensive metrics instrumentation to HotblocksDB to track HTTP API performance, stream statistics, and query execution metrics. The implementation introduces Prometheus-compatible metrics for monitoring query throughput, data transfer rates, and API endpoint usage.

Changes:

Added HTTP middleware to track request metrics including status codes, time-to-first-byte, and endpoint-specific labels
Implemented stream and query statistics tracking with histograms for bytes, blocks, chunks, and throughput rates
Integrated metrics reporter that periodically updates active query gauges

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
crates/hotblocks/src/metrics.rs	Added new metric definitions for HTTP, stream, and query tracking; reorganized registry with lazy_static
crates/hotblocks/src/query/executor.rs	Added metrics reporter for active queries and completed query counter
crates/hotblocks/src/query/running.rs	Implemented RunningQueryStats to track chunks, blocks, and bytes processed per query
crates/hotblocks/src/query/response.rs	Added QueryStreamStats to track stream metrics with time limit support and metrics reporting on drop
crates/hotblocks/src/query/service.rs	Exposed spawn_metrics_reporter method for periodic metrics updates
crates/hotblocks/src/api.rs	Added middleware for HTTP request tracking, ResponseWithMetadata pattern for endpoint labeling, and 404 handler
crates/hotblocks/src/cli.rs	Initialize metrics reporter with 5-second interval
crates/hotblocks/src/dataset_config.rs	Added clarifying comments to RetentionConfig variants
crates/hotblocks/Cargo.toml	Added lazy_static and tower-http dependencies
Cargo.lock	Updated lock file with new dependencies

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/hotblocks/src/metrics.rs

crates/hotblocks/src/api.rs

crates/hotblocks/src/metrics.rs

crates/hotblocks/src/query/response.rs

crates/hotblocks/src/query/service.rs

crates/hotblocks/src/query/running.rs

crates/hotblocks/src/query/response.rs

crates/hotblocks/src/metrics.rs

crates/hotblocks/src/api.rs

chore(NET-112): Add http api, streams and query related metrics to Ho…

3ab696c

…tblockDB

define-null marked this pull request as ready for review January 14, 2026 08:00

define-null requested a review from kalabukdima January 14, 2026 08:18

kalabukdima requested changes Jan 14, 2026

View reviewed changes

Rename dataset_id to dataset_name

b33d38e

define-null requested a review from kalabukdima January 14, 2026 12:46

mo4islona requested a review from Copilot January 14, 2026 13:47

Copilot started reviewing on behalf of mo4islona January 14, 2026 13:47 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

define-null commented Jan 14, 2026

View reviewed changes

crates/hotblocks/src/api.rs Outdated Show resolved Hide resolved

Update crates/hotblocks/src/api.rs

e98ba2d

define-null commented Jan 14, 2026

View reviewed changes

crates/hotblocks/src/api.rs Outdated Show resolved Hide resolved

define-null added 2 commits January 14, 2026 22:12

Update crates/hotblocks/src/api.rs

2ac1c62

Dropped query_bytes

84de309

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore(NET-112): Add http api, streams and query related metrics to HotblockDB #48

chore(NET-112): Add http api, streams and query related metrics to HotblockDB #48

define-null commented Jan 14, 2026 •

edited

Loading

Uh oh!

kalabukdima Jan 14, 2026

Uh oh!

define-null Jan 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chore(NET-112): Add http api, streams and query related metrics to HotblockDB #48

Are you sure you want to change the base?

chore(NET-112): Add http api, streams and query related metrics to HotblockDB #48

Conversation

define-null commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR about?

Uh oh!

kalabukdima Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

define-null Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

define-null commented Jan 14, 2026 •

edited

Loading

define-null Jan 14, 2026 •

edited

Loading